feat: add wms_oracle community connector#26
Open
fivetran-clgritton wants to merge 20 commits into
Open
Conversation
Adds a generalized Oracle WMS REST API connector syncing 26 warehouse entities with two-phase incremental sync, descending historical backfill across rolling 30-day windows, pre-cursor hourly drift detection, and parallel entity processing via ThreadPoolExecutor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Require https:// in validate_configuration (was accepting http://) - Remove dead code: apply_lookback(), INCREMENTAL_LOOKBACK_MINUTES, timedelta import - Remove lag_minutes throughout (config, code, docs) — default was 0, adds confusion - Fix page_size default inconsistency: configuration.json now shows 1000 to match utils.py and README - Delete requirements.txt — requests is pre-installed in the SDK runtime - Fix INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS formatting (Black parens artifact) - README: expand Data handling with mod_ts detection mechanics and adding-entity instructions - README: add Pre-cursor drift check section with full mechanics explanation - README: expand monitoring tables with column-level documentation and usage notes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unused datetime imports (timezone, timedelta) from connector.py introduced when lag_minutes was removed - Wrap all E501 log message f-strings across api.py, backfill.py, connector.py, incremental.py, and pre_sync_drift_check.py - Apply black reformatting to utils.py (INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Black always generates W503-style line breaks (operator at start of continuation line). Without this, any connector using multi-line boolean conditions will fail the linting check. W503 and W504 are mutually exclusive — suppressing W503 is the standard fix for Black + flake8 coexistence. Also removes noqa: W503 comments added as a temporary workaround. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The comments were adding enough length to trigger Black reformatting. W503 is now globally ignored via .flake8 so the comments are redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new wms_oracle/ community connector example that syncs Oracle WMS Cloud REST API entities into Fivetran, including incremental sync + historical backfill logic and pre-sync drift monitoring, and wires it into the repository documentation.
Changes:
- Added a full Oracle WMS connector implementation split across
connector.pyand helper modules (api.py,incremental.py,backfill.py,pre_sync_drift_check.py,utils.py). - Added connector documentation (
wms_oracle/README.md) and example config (wms_oracle/configuration.json). - Updated repository-level docs/linting (
README.md,.flake8).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 19 comments.
Show a summary per file
| File | Description |
|---|---|
| wms_oracle/connector.py | Main connector entrypoint, schema, and sync orchestration (incremental/backfill/drift-check). |
| wms_oracle/api.py | Oracle WMS REST API request + pagination utilities and capability probing. |
| wms_oracle/incremental.py | Two-phase incremental sync implementation (mod_ts + create_ts catch-up). |
| wms_oracle/backfill.py | Historical backfill implementation using rolling windows and DESC pagination. |
| wms_oracle/pre_sync_drift_check.py | Pre-sync hourly drift detection and monitoring table writes. |
| wms_oracle/utils.py | Constants, entity list, configuration validation, timestamp utilities. |
| wms_oracle/configuration.json | Example connector configuration template. |
| wms_oracle/README.md | Connector example documentation. |
| README.md | Adds wms_oracle to the catalog list. |
| .flake8 | Updates global flake8 ignore list. |
fivetran-JenasVimal
requested changes
Jun 22, 2026
fivetran-JenasVimal
left a comment
Contributor
There was a problem hiding this comment.
Please Address all of the copilot comments
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Add template-style import comments to connector.py and api.py
- Replace update() docstring with verbatim template wording
- Add log.warning("Example: Oracle WMS : wms_oracle") as first log in update()
- Replace single-line upsert comment with full template comment block
- Add template checkpoint comment block before op.checkpoint()
- Add template __main__ block comments
- Add full Description/Args/Returns docstrings to check_entity_has_mod_ts and probe_entity_count
- Add retry logic (408, 429, 500-504, Timeout) to check_entity_has_mod_ts
- Replace literal defaults in configuration.json and README with angle-bracket placeholders
- Add numbered credential-setup steps to README Authentication section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add full Description/Args/Returns/Raises to make_api_request and fetch_entity_data - Refactor multi-line boolean conditions into intermediate variables to eliminate W503 (line-break-before-binary-operator) violations without flake8 suppression Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace chained variable assignments with clearer named variables and explicit break conditions, eliminating W503 violations without flake8 suppression. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nt log - Refactor multi-line boolean conditions in backfill.py and incremental.py into named variables to eliminate W503 without flake8 suppression - Simplify string concatenation in connector.py log statement - Remove redundant log.info after log.warning example banner in update() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckfill Replace two-line ts_boundary_cleared assignment with a single readable line using short-circuit evaluation, and rename to crossed_ts_boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… by Pass 1 All incremental_only entities have an incremental cursor (required by classification), so they are always included in entities_with_cursor and processed in Pass 1. Pass 3 could never execute. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Worker threads in the backfill pass now do API fetching only, putting upsert/truncate/checkpoint messages on a queue.Queue. The main thread drains the queue and performs all op.* calls, complying with the SDK's single-threaded output stream requirement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… error
("checkpoint", state) and ("truncate", entity) were 2-tuples but the
drain loop always unpacked 3 values, causing ValueError at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Guard against any unexpected queue message format with a warning log before unpacking. If a non-3-tuple arrives it is skipped and logged rather than crashing the sync, so the next run reveals the exact message that caused the original ValueError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_has_mod_ts Returning False after all retries fail was silently caching an incorrect mod_ts capability flag in state, causing repeated unnecessary full scans and op.truncate() calls. Now raises on transient failures (408, 429, 5xx, Timeout) so the sync fails cleanly instead of poisoning the cache. Permanent errors (non-transient status codes) still return False. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Permanent errors (auth failures, connection errors, non-transient HTTP status codes) now also raise instead of returning False, so no describe failure can silently cache an incorrect mod_ts capability in state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ConnectionError has no HTTP status code (status=None), so it bypassed the transient_status_codes check and raised immediately without retrying. Now any RequestException without a status code is treated as transient and retried like 408/429/5xx. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wms_oracle/— a generalized Oracle WMS Cloud REST API connector syncing 26 warehouse entities (orders, inventory, containers, purchasing documents, and more)wms_oracle/README.mdwith full documentation following the repo templateREADME.mdto listwms_oracleunder SaaS & APIsConnector highlights
mod_tscursor-advancement (Phase 1) +create_tscatch-up for backdated records (Phase 2)ThreadPoolExecutorfor backfill andmod_tscapability discoverypage_sizeon timeout and recalculates offset to resume without data losscounts_by_day,pre_cursor_hourly_counts) written each sync for observabilityTest plan
fivetran debug --configuration=configuration.jsonwith a real Oracle WMS instance and confirmwarehouse.dbis populatedgrep -ri "simfoods\|simmons")flake8passes (W503 only, pre-existing repo-wide)black --checkpasses🤖 Generated with Claude Code